This notebook will report the findings of creating a decision tree model on a binary version of the Boston housing dataset. For reference:

  1. CRIM: per capita crime rate by town
  2. ZN: proportion of residential land zoned for lots over 25,000 sq.ft .
  3. INDUS: proportion of non retail business acres per town
  4. CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
  5. NOX: nitric oxides concentration (parts per 10 million)
  6. RM: average number of rooms per dwelling
  7. AGE: proportion of owner occupied units built prior to 1940
  8. DIS: weighted distances to five Boston employment centers
  9. RAD: index of accessibility to radial highways
  10. TAX: full value property tax rate per 10,000
  11. PTRATIO: pupil teacher ratio by town
  12. B: 1000(Bk 0.63)^2 where Bk is the proportion of blacks by town
  13. LSTAT: percent lower status of the population
  14. MEDV: Median value of owner occupied homes in 1000's

Note that in this dataset, an instance has a median house value of 1 if the median house value is greater than 230k and 0 if it is not.

First we must load the dataset.

In [1]:
housing<-read.table("housing.header.binary.txt", header=TRUE, sep=",")
colnames(housing)<-c("Crim", "Zn", "Indus", "Chas", "Nox", "Rm", "Age", "Dis", "Rad", "Tax", "Ptratio", "B", "Lstat", "Medv")
summary(housing)

      Crim                Zn             Indus            Chas        
 Min.   : 0.00632   Min.   :  0.00   Min.   : 0.46   Min.   :0.00000  
 1st Qu.: 0.08204   1st Qu.:  0.00   1st Qu.: 5.19   1st Qu.:0.00000  
 Median : 0.25651   Median :  0.00   Median : 9.69   Median :0.00000  
 Mean   : 3.61352   Mean   : 11.36   Mean   :11.14   Mean   :0.06917  
 3rd Qu.: 3.67708   3rd Qu.: 12.50   3rd Qu.:18.10   3rd Qu.:0.00000  
 Max.   :88.97620   Max.   :100.00   Max.   :27.74   Max.   :1.00000  
      Nox               Rm             Age              Dis        
 Min.   :0.3850   Min.   :3.561   Min.   :  2.90   Min.   : 1.130  
 1st Qu.:0.4490   1st Qu.:5.886   1st Qu.: 45.02   1st Qu.: 2.100  
 Median :0.5380   Median :6.208   Median : 77.50   Median : 3.207  
 Mean   :0.5547   Mean   :6.285   Mean   : 68.57   Mean   : 3.795  
 3rd Qu.:0.6240   3rd Qu.:6.623   3rd Qu.: 94.08   3rd Qu.: 5.188  
 Max.   :0.8710   Max.   :8.780   Max.   :100.00   Max.   :12.127  
      Rad              Tax           Ptratio            B         
 Min.   : 1.000   Min.   :187.0   Min.   :12.60   Min.   :  0.32  
 1st Qu.: 4.000   1st Qu.:279.0   1st Qu.:17.40   1st Qu.:375.38  
 Median : 5.000   Median :330.0   Median :19.05   Median :391.44  
 Mean   : 9.549   Mean   :408.2   Mean   :18.46   Mean   :356.67  
 3rd Qu.:24.000   3rd Qu.:666.0   3rd Qu.:20.20   3rd Qu.:396.23  
 Max.   :24.000   Max.   :711.0   Max.   :22.00   Max.   :396.90  
     Lstat            Medv       
 Min.   : 1.73   Min.   :0.0000  
 1st Qu.: 6.95   1st Qu.:0.0000  
 Median :11.36   Median :0.0000  
 Mean   :12.65   Mean   :0.3834  
 3rd Qu.:16.95   3rd Qu.:1.0000  
 Max.   :37.97   Max.   :1.0000

We will use the Crim index and number of rooms in a building as independent variables to train an RPART decision tree that predicts whether the median house value is 0 or 1.

In [2]:
library(rpart)
fit<-rpart(Medv~Crim+Rm, method="class", data=housing)
summary(fit)

Call:
rpart(formula = Medv ~ Crim + Rm, data = housing, method = "class")
  n= 506 

          CP nsplit rel error    xerror       xstd
1 0.51546392      0 1.0000000 1.0000000 0.05637690
2 0.05670103      1 0.4845361 0.5206186 0.04634589
3 0.01546392      2 0.4278351 0.4639175 0.04433951
4 0.01374570      5 0.3814433 0.5257732 0.04651722
5 0.01030928      8 0.3402062 0.5103093 0.04599788
6 0.01000000     11 0.3041237 0.5154639 0.04617278

Variable importance
  Rm Crim 
  69   31 

Node number 1: 506 observations,    complexity param=0.5154639
  predicted class=0  expected loss=0.3833992  P(node) =1
    class counts:   312   194
   probabilities: 0.617 0.383 
  left son=2 (362 obs) right son=3 (144 obs)
  Primary splits:
      Rm   < 6.5455   to the left,  improve=86.60421, (0 missing)
      Crim < 0.66771  to the right, improve=30.94264, (0 missing)
  Surrogate splits:
      Crim < 0.0273   to the right, agree=0.739, adj=0.083, (0 split)

Node number 2: 362 observations,    complexity param=0.01546392
  predicted class=0  expected loss=0.198895  P(node) =0.715415
    class counts:   290    72
   probabilities: 0.801 0.199 
  left son=4 (227 obs) right son=5 (135 obs)
  Primary splits:
      Rm   < 6.143    to the left,  improve=10.567340, (0 missing)
      Crim < 0.60872  to the right, improve= 5.439062, (0 missing)
  Surrogate splits:
      Crim < 0.042975 to the right, agree=0.652, adj=0.067, (0 split)

Node number 3: 144 observations,    complexity param=0.05670103
  predicted class=1  expected loss=0.1527778  P(node) =0.284585
    class counts:    22   122
   probabilities: 0.153 0.847 
  left son=6 (13 obs) right son=7 (131 obs)
  Primary splits:
      Crim < 6.62824  to the right, improve=16.958340, (0 missing)
      Rm   < 6.8375   to the left,  improve= 3.992841, (0 missing)

Node number 4: 227 observations
  predicted class=0  expected loss=0.1057269  P(node) =0.4486166
    class counts:   203    24
   probabilities: 0.894 0.106 

Node number 5: 135 observations,    complexity param=0.01546392
  predicted class=0  expected loss=0.3555556  P(node) =0.2667984
    class counts:    87    48
   probabilities: 0.644 0.356 
  left son=10 (47 obs) right son=11 (88 obs)
  Primary splits:
      Crim < 3.621175 to the right, improve=12.2723700, (0 missing)
      Rm   < 6.365    to the left,  improve= 0.9333923, (0 missing)

Node number 6: 13 observations
  predicted class=0  expected loss=0.07692308  P(node) =0.0256917
    class counts:    12     1
   probabilities: 0.923 0.077 

Node number 7: 131 observations
  predicted class=1  expected loss=0.07633588  P(node) =0.2588933
    class counts:    10   121
   probabilities: 0.076 0.924 

Node number 10: 47 observations
  predicted class=0  expected loss=0.06382979  P(node) =0.09288538
    class counts:    44     3
   probabilities: 0.936 0.064 

Node number 11: 88 observations,    complexity param=0.01546392
  predicted class=1  expected loss=0.4886364  P(node) =0.173913
    class counts:    43    45
   probabilities: 0.489 0.511 
  left son=22 (47 obs) right son=23 (41 obs)
  Primary splits:
      Rm   < 6.365    to the left,  improve=1.486354, (0 missing)
      Crim < 0.21949  to the right, improve=1.227273, (0 missing)
  Surrogate splits:
      Crim < 0.02803  to the right, agree=0.58, adj=0.098, (0 split)

Node number 22: 47 observations,    complexity param=0.0137457
  predicted class=0  expected loss=0.4255319  P(node) =0.09288538
    class counts:    27    20
   probabilities: 0.574 0.426 
  left son=44 (39 obs) right son=45 (8 obs)
  Primary splits:
      Crim < 0.040825 to the right, improve=0.7671849, (0 missing)
      Rm   < 6.3155   to the right, improve=0.5941080, (0 missing)

Node number 23: 41 observations,    complexity param=0.01030928
  predicted class=1  expected loss=0.3902439  P(node) =0.08102767
    class counts:    16    25
   probabilities: 0.390 0.610 
  left son=46 (32 obs) right son=47 (9 obs)
  Primary splits:
      Rm   < 6.3975   to the right, improve=1.796917, (0 missing)
      Crim < 0.048715 to the left,  improve=1.762195, (0 missing)

Node number 44: 39 observations,    complexity param=0.0137457
  predicted class=0  expected loss=0.3846154  P(node) =0.0770751
    class counts:    24    15
   probabilities: 0.615 0.385 
  left son=88 (18 obs) right son=89 (21 obs)
  Primary splits:
      Crim < 0.14055  to the left,  improve=1.7631260, (0 missing)
      Rm   < 6.3155   to the right, improve=0.3647643, (0 missing)
  Surrogate splits:
      Rm < 6.2695   to the right, agree=0.641, adj=0.222, (0 split)

Node number 45: 8 observations
  predicted class=1  expected loss=0.375  P(node) =0.01581028
    class counts:     3     5
   probabilities: 0.375 0.625 

Node number 46: 32 observations,    complexity param=0.01030928
  predicted class=1  expected loss=0.46875  P(node) =0.06324111
    class counts:    15    17
   probabilities: 0.469 0.531 
  left son=92 (8 obs) right son=93 (24 obs)
  Primary splits:
      Crim < 0.05022  to the left,  improve=1.687500, (0 missing)
      Rm   < 6.4775   to the left,  improve=1.288149, (0 missing)
  Surrogate splits:
      Rm < 6.5135   to the right, agree=0.781, adj=0.125, (0 split)

Node number 47: 9 observations
  predicted class=1  expected loss=0.1111111  P(node) =0.01778656
    class counts:     1     8
   probabilities: 0.111 0.889 

Node number 88: 18 observations
  predicted class=0  expected loss=0.2222222  P(node) =0.03557312
    class counts:    14     4
   probabilities: 0.778 0.222 

Node number 89: 21 observations,    complexity param=0.0137457
  predicted class=1  expected loss=0.4761905  P(node) =0.04150198
    class counts:    10    11
   probabilities: 0.476 0.524 
  left son=178 (11 obs) right son=179 (10 obs)
  Primary splits:
      Crim < 0.24666  to the right, improve=2.912554, (0 missing)
      Rm   < 6.2345   to the left,  improve=0.221645, (0 missing)
  Surrogate splits:
      Rm < 6.2475   to the right, agree=0.667, adj=0.3, (0 split)

Node number 92: 8 observations
  predicted class=0  expected loss=0.25  P(node) =0.01581028
    class counts:     6     2
   probabilities: 0.750 0.250 

Node number 93: 24 observations,    complexity param=0.01030928
  predicted class=1  expected loss=0.375  P(node) =0.04743083
    class counts:     9    15
   probabilities: 0.375 0.625 
  left son=186 (9 obs) right son=187 (15 obs)
  Primary splits:
      Crim < 0.221425 to the right, improve=2.4500000, (0 missing)
      Rm   < 6.4775   to the left,  improve=0.6722222, (0 missing)
  Surrogate splits:
      Rm < 6.4155   to the left,  agree=0.75, adj=0.333, (0 split)

Node number 178: 11 observations
  predicted class=0  expected loss=0.2727273  P(node) =0.02173913
    class counts:     8     3
   probabilities: 0.727 0.273 

Node number 179: 10 observations
  predicted class=1  expected loss=0.2  P(node) =0.01976285
    class counts:     2     8
   probabilities: 0.200 0.800 

Node number 186: 9 observations
  predicted class=0  expected loss=0.3333333  P(node) =0.01778656
    class counts:     6     3
   probabilities: 0.667 0.333 

Node number 187: 15 observations
  predicted class=1  expected loss=0.2  P(node) =0.02964427
    class counts:     3    12
   probabilities: 0.200 0.800

Next we visualize the tree:

In [3]:
plot(fit, uniform=TRUE, main="Decision Tree")
text(fit, use.n=TRUE, all=TRUE, cex=.8)

The above decision tree looks at each instance and performs the following splits:

Root node-Is Rm<6.546?

312 instances were split as negatives, and 194 were split as positives.

Traversing the left subtree, we ask:

Is Rm < 6.143?

290 instances were split as negatives, and 72 were split as positives.

Continuing down the left subtree, we have a class label of 0 in response to an answer of 0 to the above question. Here we have 203 instances that were true negatives and 24 that were not.

Traversing the right subtree from the above node, we ask:

Is Crim>=3.621?

87 instances were split as negatives, and 48 were split as positives.

Moving left from here, we have a class label of 0 in response to an answer of 0 to the above question. Here we have 44 instances that were true negatives and 3 that were not.

Traversing the right subtree from the above node, we ask:

Is Rm > 6.365?

43 instances were split as negatives, and 45 were split as positives.

Traversing the left subtree from this node, we ask:

Is Crim >=0.0483?

27 instances were split as negatives, and 20 were split as positives.

Moving right from here, we have a class label of 1 in response to an answer of 1 to the above question. Here we have 5 instances that were true positives and 3 that were not.

The above described process can be used for each decision node to eventually reach a leaf node.

Now we will use all variables to train an RPART decision tree that predicts whether the median house value is 0 or 1. We will use 80% of the data to fit the tree, and 20% of the data to test the effectiveness of the decision tree model.

In [4]:
set.seed(1)
train<-sample(1:nrow(housing), 0.8*nrow(housing))
train.set<-housing[train,]
test.set<-housing[-train,]
housing.tree<-rpart(Medv~Crim+Zn+Indus+Chas+Nox+Rm+Age+Dis+Rad+Tax+Ptratio+B+Lstat, data=housing[train,], method="class")
summary(housing.tree)

Call:
rpart(formula = Medv ~ Crim + Zn + Indus + Chas + Nox + Rm + 
    Age + Dis + Rad + Tax + Ptratio + B + Lstat, data = housing[train, 
    ], method = "class")
  n= 404 

          CP nsplit rel error    xerror       xstd
1 0.65644172      0 1.0000000 1.0000000 0.06049563
2 0.06134969      1 0.3435583 0.4233129 0.04640542
3 0.03067485      2 0.2822086 0.4110429 0.04586423
4 0.01000000      3 0.2515337 0.3742331 0.04415030

Variable importance
Lstat    Rm Indus   Nox  Crim   Age   Dis     B 
   25    17    16    15    13    13     1     1 

Node number 1: 404 observations,    complexity param=0.6564417
  predicted class=0  expected loss=0.4034653  P(node) =1
    class counts:   241   163
   probabilities: 0.597 0.403 
  left son=2 (227 obs) right son=3 (177 obs)
  Primary splits:
      Lstat   < 10.14    to the right, improve=100.19760, (0 missing)
      Rm      < 6.5455   to the left,  improve= 72.03598, (0 missing)
      Indus   < 8.005    to the right, improve= 51.39322, (0 missing)
      Ptratio < 19.9     to the right, improve= 45.67274, (0 missing)
      Nox     < 0.5125   to the right, improve= 45.53887, (0 missing)
  Surrogate splits:
      Indus < 7.625    to the right, agree=0.829, adj=0.610, (0 split)
      Nox   < 0.519    to the right, agree=0.809, adj=0.565, (0 split)
      Rm    < 6.476    to the left,  agree=0.790, adj=0.520, (0 split)
      Age   < 52.05    to the right, agree=0.775, adj=0.486, (0 split)
      Crim  < 0.09278  to the right, agree=0.767, adj=0.469, (0 split)

Node number 2: 227 observations
  predicted class=0  expected loss=0.09251101  P(node) =0.5618812
    class counts:   206    21
   probabilities: 0.907 0.093 

Node number 3: 177 observations,    complexity param=0.06134969
  predicted class=1  expected loss=0.1977401  P(node) =0.4381188
    class counts:    35   142
   probabilities: 0.198 0.802 
  left son=6 (30 obs) right son=7 (147 obs)
  Primary splits:
      Rm      < 6.1445   to the left,  improve=15.886080, (0 missing)
      Lstat   < 7.765    to the right, improve=12.974080, (0 missing)
      Ptratio < 20.95    to the right, improve= 3.889285, (0 missing)
      Rad     < 4.5      to the left,  improve= 3.498422, (0 missing)
      Crim    < 0.073735 to the left,  improve= 2.754341, (0 missing)
  Surrogate splits:
      Lstat   < 9.725    to the right, agree=0.864, adj=0.200, (0 split)
      Ptratio < 20.95    to the right, agree=0.847, adj=0.100, (0 split)
      B       < 354.1    to the left,  agree=0.847, adj=0.100, (0 split)
      Dis     < 1.7616   to the left,  agree=0.842, adj=0.067, (0 split)
      Crim    < 7.403005 to the right, agree=0.836, adj=0.033, (0 split)

Node number 6: 30 observations,    complexity param=0.03067485
  predicted class=0  expected loss=0.3333333  P(node) =0.07425743
    class counts:    20    10
   probabilities: 0.667 0.333 
  left son=12 (19 obs) right son=13 (11 obs)
  Primary splits:
      Crim  < 0.166905 to the left,  improve=5.390750, (0 missing)
      Indus < 12.925   to the left,  improve=5.010352, (0 missing)
      Dis   < 4.60305  to the right, improve=4.444444, (0 missing)
      Rad   < 4.5      to the left,  improve=2.976190, (0 missing)
      Nox   < 0.5555   to the left,  improve=2.650104, (0 missing)
  Surrogate splits:
      Nox   < 0.5265   to the left,  agree=0.900, adj=0.727, (0 split)
      Dis   < 4.75985  to the right, agree=0.867, adj=0.636, (0 split)
      Indus < 16.01    to the left,  agree=0.833, adj=0.545, (0 split)
      Age   < 51.85    to the left,  agree=0.833, adj=0.545, (0 split)
      B     < 358.475  to the right, agree=0.800, adj=0.455, (0 split)

Node number 7: 147 observations
  predicted class=1  expected loss=0.1020408  P(node) =0.3638614
    class counts:    15   132
   probabilities: 0.102 0.898 

Node number 12: 19 observations
  predicted class=0  expected loss=0.1052632  P(node) =0.0470297
    class counts:    17     2
   probabilities: 0.895 0.105 

Node number 13: 11 observations
  predicted class=1  expected loss=0.2727273  P(node) =0.02722772
    class counts:     3     8
   probabilities: 0.273 0.727
In [5]:
housing.pred<-predict(housing.tree, housing[-train,], type="class")
table(housing[-train,]$Medv, housing.pred)

   housing.pred
     0  1
  0 62  9
  1  7 24

From this, we have that:

True Positive Rate=24/(24+7)=24/31=0.7742

False Positive Rate=9/(9+62)=9/71=0.1268

Accuracy: (62+24)/(62+24+9+7)=86/102=84.31%

Next we show the ROC curve of the classifier which plots true positve rate as a function of false positve rate:

In [6]:
housing.prob<-predict(housing.tree,housing[-train, ], type="prob")
housing.prob.label<-cbind(housing.prob[, 2], housing[-train,]$Medv)
library(ROCR)

pred<-prediction(housing.prob.label[, 1], housing.prob.label[, 2])
perf<-performance(pred, "tpr", "fpr")
plot(perf, col="red")
abline(0,1, col="lightgray")

Loading required package: gplots

Attaching package: 'gplots'

The following object is masked from 'package:stats':

    lowess

Next we show the AUC value:

In [8]:
auc<-performance(pred, "auc")
auc
auc@y.values[[1]]

An object of class "performance"
Slot "x.name":
[1] "None"

Slot "y.name":
[1] "Area under the ROC curve"

Slot "alpha.name":
[1] "none"

Slot "x.values":
list()

Slot "y.values":
[[1]]
[1] 0.8523398


Slot "alpha.values":
list()
0.852339845524761

Next we can create a new instance on which to perform a prediction:

In [9]:
ninst<-c(0.03, 13, 3.5, 0.3, 0.58, 4.1, 68, 4.98, 3, 225, 17, 396, 7.56)
names(ninst)<-c("Crim", "Zn", "Indus", "Chas", "Nox", "Rm", "Age", "Dis", "Rad", "Tax", "Ptratio", "B", "Lstat")
ninst.frame<-data.frame(ninst)
plot(housing.tree, uniform=TRUE, main="Decision Tree")
text(housing.tree, use.n=TRUE, all=TRUE, cex=.8)

Using this decison tree on the new instance, we get a class label of 0.